Backpropagation Neural Network Architecture 

The BP network implements the generalized delta rule. It is a gradient 
descent algorithm which minimizes the squared error of the network. The 
gradient descent algorithm is applied to adjust the connected weights. 

The training process of the BP neural network generally involves five steps: 

1. Select representative training samples and turn them into the input 
layer as the input value. 

2. Calculate the predictive value of the network. 

3. Compare the target value with the predictive value to obtain the error 
value. 

4. Readjust the weights in each layer of the network according to the 
error value. 

5. Repeat the above procedure until the error value of each training 
sample is minimized, meaning that the training is finished. 

Gradient Descent 
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The following is the scenario: 

The output of the kth output neuron, s k is: 

s k = f(yk) 

where y k is the network input to the kth output neuron which is written as: 
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where /(z, ) is the output of the jth hidden unit 

where Zj is the network input to the /th hidden neuron which is written as: 

z j =^ j x i w ij 
i 

where is /th input unit and w t j is the weights of the hidden layer. 
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The error function to be minimized is: 
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where d is the desired output vector. 

for weight connected from hidden to output units: 


dE 


dE df(y k ) dy k 


dwjk df(y k ) dy k dw jk 

where w ]k is the weight of the output layer. 

let f(y k ) = sigmoidal function 


dE dy k 

= -(d- /(y fc ))/(y fc )( 1 “ f(yO) 


dw jk 


dw jk 


dE 

dw jk 


= -e /(y fc )( l - /(y fe )) f(zj) 


Let S k = e /(y fe )( 1 - /(y fc )) 

dE 


dw jk 


= -S k f{zj) 


for weight connected from input to hidden units: 


dE /y d_E_ d/(y fc ) dy k \ dfjzf) dzj 

dw ij \2Ldf(y k ) dy k df(Zj) J dzj dw tj 


^ -e /(y fc )(l - /(y fc )) w Jk j f(zjX 1 - /O;)) x i 
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Let 


^~. = ~i^ Sk fW 1 ~ Xi 

= ^ S k W,*) /(Zy)(l - /(Zy)) 
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dE 


dw. 
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the change in weights of the output units are given by 


Aw /fc = -r\ ■ 


dE 


Vjk ' dw jk 


= r}8 k f(zj) 


where 77 learning rate parameter. 

The change in weights of the hidden units is 


A Wn = —r] 


dE 

dwij 


= V 8j x t 


The weights update equations are given in 
w jk new = w jk old + Aw jk 

W ..new _ w ..old + fr w .. 


where Wji new and Wji new are the new weights, Wj° ld and Wj° ld are the 
previous weights. 
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V = /dO’46 v 4 + I'se.Vs) 
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